Summary

Examining the impact of cognitive load on structure learning

6 agents, 12 issues

Method changes:

  • ?
Demographics (Attention Check)
0
0.25
0.5
0.75
1
Overall
high
(N=52)
low
(N=37)
high
(N=41)
low
(N=54)
high
(N=49)
low
(N=51)
high
(N=41)
low
(N=47)
high
(N=45)
low
(N=41)
high
(N=228)
low
(N=230)
age
Mean (SD) 36.3 (11.7) 38.4 (13.1) 37.3 (12.8) 39.4 (13.2) 38.8 (13.9) 37.0 (13.9) 37.3 (9.68) 36.1 (12.2) 38.1 (12.7) 36.9 (11.0) 37.6 (12.2) 37.6 (12.7)
Median [Min, Max] 33.5 [19.0, 60.0] 37.0 [22.0, 67.0] 33.0 [18.0, 68.0] 36.5 [18.0, 67.0] 37.0 [18.0, 70.0] 34.0 [19.0, 75.0] 36.0 [21.0, 56.0] 36.0 [19.0, 80.0] 34.0 [19.0, 67.0] 37.0 [20.0, 63.0] 34.5 [18.0, 70.0] 36.0 [18.0, 80.0]
race
American Indian or Alaska Native 3 (5.8%) 0 (0%) 0 (0%) 1 (1.9%) 0 (0%) 0 (0%) 1 (2.4%) 0 (0%) 0 (0%) 1 (2.4%) 4 (1.8%) 2 (0.9%)
Asian 4 (7.7%) 5 (13.5%) 4 (9.8%) 4 (7.4%) 3 (6.1%) 4 (7.8%) 4 (9.8%) 2 (4.3%) 6 (13.3%) 3 (7.3%) 21 (9.2%) 18 (7.8%)
Black or African-American 8 (15.4%) 7 (18.9%) 6 (14.6%) 11 (20.4%) 9 (18.4%) 8 (15.7%) 5 (12.2%) 8 (17.0%) 9 (20.0%) 5 (12.2%) 37 (16.2%) 39 (17.0%)
Hispanic/Latinx 2 (3.8%) 3 (8.1%) 5 (12.2%) 9 (16.7%) 5 (10.2%) 4 (7.8%) 4 (9.8%) 4 (8.5%) 3 (6.7%) 3 (7.3%) 19 (8.3%) 23 (10.0%)
Native Hawaiian or Other Pacific Islander 1 (1.9%) 0 (0%) 0 (0%) 0 (0%) 0 (0%) 0 (0%) 0 (0%) 0 (0%) 0 (0%) 0 (0%) 1 (0.4%) 0 (0%)
White 34 (65.4%) 22 (59.5%) 26 (63.4%) 29 (53.7%) 30 (61.2%) 35 (68.6%) 27 (65.9%) 32 (68.1%) 27 (60.0%) 28 (68.3%) 144 (63.2%) 146 (63.5%)
Other 0 (0%) 0 (0%) 0 (0%) 0 (0%) 2 (4.1%) 0 (0%) 0 (0%) 1 (2.1%) 0 (0%) 1 (2.4%) 2 (0.9%) 2 (0.9%)
gender
Man 23 (44.2%) 15 (40.5%) 18 (43.9%) 21 (38.9%) 20 (40.8%) 27 (52.9%) 19 (46.3%) 22 (46.8%) 16 (35.6%) 12 (29.3%) 96 (42.1%) 97 (42.2%)
Non-binary 2 (3.8%) 1 (2.7%) 1 (2.4%) 2 (3.7%) 1 (2.0%) 0 (0%) 1 (2.4%) 0 (0%) 1 (2.2%) 0 (0%) 6 (2.6%) 3 (1.3%)
Woman 27 (51.9%) 21 (56.8%) 22 (53.7%) 30 (55.6%) 26 (53.1%) 24 (47.1%) 21 (51.2%) 25 (53.2%) 28 (62.2%) 29 (70.7%) 124 (54.4%) 129 (56.1%)
Prefer not to answer 0 (0%) 0 (0%) 0 (0%) 1 (1.9%) 2 (4.1%) 0 (0%) 0 (0%) 0 (0%) 0 (0%) 0 (0%) 2 (0.9%) 1 (0.4%)
matrix_acc
Mean (SD) 0.750 (0.269) 0.956 (0.0735) 0.771 (0.259) 0.896 (0.197) 0.829 (0.165) 0.914 (0.163) 0.787 (0.190) 0.955 (0.0800) 0.739 (0.199) 0.954 (0.0917) 0.775 (0.221) 0.932 (0.138)
Median [Min, Max] 0.875 [0, 1.00] 1.00 [0.750, 1.00] 0.875 [0, 1.00] 1.00 [0, 1.00] 0.875 [0.250, 1.00] 1.00 [0, 1.00] 0.750 [0.375, 1.00] 1.00 [0.625, 1.00] 0.750 [0.125, 1.00] 1.00 [0.625, 1.00] 0.875 [0, 1.00] 1.00 [0, 1.00]
as.factor(matrix_n_correct)
0 3 (5.8%) 0 (0%) 2 (4.9%) 2 (3.7%) 0 (0%) 1 (2.0%) 0 (0%) 0 (0%) 0 (0%) 0 (0%) 5 (2.2%) 3 (1.3%)
1 0 (0%) 0 (0%) 1 (2.4%) 0 (0%) 0 (0%) 0 (0%) 0 (0%) 0 (0%) 1 (2.2%) 0 (0%) 2 (0.9%) 0 (0%)
2 1 (1.9%) 0 (0%) 0 (0%) 0 (0%) 1 (2.0%) 0 (0%) 0 (0%) 0 (0%) 0 (0%) 0 (0%) 2 (0.9%) 0 (0%)
3 3 (5.8%) 0 (0%) 1 (2.4%) 0 (0%) 0 (0%) 0 (0%) 4 (9.8%) 0 (0%) 2 (4.4%) 0 (0%) 10 (4.4%) 0 (0%)
4 1 (1.9%) 0 (0%) 2 (4.9%) 0 (0%) 3 (6.1%) 0 (0%) 0 (0%) 0 (0%) 5 (11.1%) 0 (0%) 11 (4.8%) 0 (0%)
5 9 (17.3%) 0 (0%) 2 (4.9%) 0 (0%) 2 (4.1%) 1 (2.0%) 7 (17.1%) 1 (2.1%) 9 (20.0%) 2 (4.9%) 29 (12.7%) 4 (1.7%)
6 8 (15.4%) 2 (5.4%) 12 (29.3%) 6 (11.1%) 15 (30.6%) 6 (11.8%) 10 (24.4%) 1 (2.1%) 9 (20.0%) 0 (0%) 54 (23.7%) 15 (6.5%)
7 12 (23.1%) 9 (24.3%) 9 (22.0%) 17 (31.5%) 13 (26.5%) 12 (23.5%) 9 (22.0%) 12 (25.5%) 12 (26.7%) 9 (22.0%) 55 (24.1%) 59 (25.7%)
8 15 (28.8%) 26 (70.3%) 12 (29.3%) 29 (53.7%) 15 (30.6%) 31 (60.8%) 11 (26.8%) 33 (70.2%) 7 (15.6%) 30 (73.2%) 60 (26.3%) 149 (64.8%)
Agent Learning Plots
Analysis of Deviance Table (Type II Wald chisquare tests)

Response: corrresp
                                               Chisq Df Pr(>Chisq)    
opinion_round                               243.3307  1  < 2.2e-16 ***
Deviant_threshold                            14.4594  4   0.005964 ** 
matrix_cond                                   1.1573  1   0.282032    
opinion_round:Deviant_threshold               9.3619  4   0.052663 .  
opinion_round:matrix_cond                     0.7073  1   0.400331    
Deviant_threshold:matrix_cond                 7.5011  4   0.111660    
opinion_round:Deviant_threshold:matrix_cond   4.8297  4   0.305225    
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
 1       opinion_round.trend    SE  df asymp.LCL asymp.UCL z.ratio p.value
 overall               0.171 0.011 Inf      0.15     0.193  15.603  <.0001

Results are averaged over the levels of: Deviant_threshold, matrix_cond 
Confidence level used: 0.95 
$emmeans
 Deviant_threshold emmean     SE  df asymp.LCL asymp.UCL z.ratio p.value
 0                   1.55 0.0958 Inf     1.360      1.74  16.155  <.0001
 0.25                1.37 0.0917 Inf     1.189      1.55  14.936  <.0001
 0.5                 1.23 0.0878 Inf     1.056      1.40  13.982  <.0001
 0.75                1.14 0.0932 Inf     0.954      1.32  12.196  <.0001
 1                   1.27 0.0949 Inf     1.082      1.45  13.358  <.0001

Results are averaged over the levels of: matrix_cond 
Results are given on the logit (not the response) scale. 
Confidence level used: 0.95 

$contrasts
 contrast                                      estimate    SE  df asymp.LCL
 Deviant_threshold0 - Deviant_threshold0.25      0.1789 0.132 Inf   -0.1821
 Deviant_threshold0 - Deviant_threshold0.5       0.3198 0.130 Inf   -0.0342
 Deviant_threshold0 - Deviant_threshold0.75      0.4117 0.133 Inf    0.0475
 Deviant_threshold0 - Deviant_threshold1         0.2799 0.135 Inf   -0.0876
 Deviant_threshold0.25 - Deviant_threshold0.5    0.1409 0.127 Inf   -0.2048
 Deviant_threshold0.25 - Deviant_threshold0.75   0.2327 0.131 Inf   -0.1233
 Deviant_threshold0.25 - Deviant_threshold1      0.1010 0.132 Inf   -0.2585
 Deviant_threshold0.5 - Deviant_threshold0.75    0.0918 0.128 Inf   -0.2571
 Deviant_threshold0.5 - Deviant_threshold1      -0.0399 0.129 Inf   -0.3924
 Deviant_threshold0.75 - Deviant_threshold1     -0.1317 0.133 Inf   -0.4943
 asymp.UCL z.ratio p.value
     0.540   1.352  0.6584
     0.674   2.464  0.0989
     0.776   3.084  0.0175
     0.647   2.078  0.2296
     0.487   1.112  0.8005
     0.589   1.783  0.3834
     0.460   0.766  0.9402
     0.441   0.718  0.9525
     0.313  -0.309  0.9980
     0.231  -0.991  0.8595

Results are averaged over the levels of: matrix_cond 
Results are given on the log odds ratio (not the response) scale. 
Confidence level used: 0.95 
Conf-level adjustment: tukey method for comparing a family of 5 estimates 
P value adjustment: tukey method for comparing a family of 5 estimates 
$emmeans
 matrix_cond emmean     SE  df asymp.LCL asymp.UCL z.ratio p.value
 high          1.37 0.0588 Inf      1.25      1.48  23.222  <.0001
 low           1.25 0.0587 Inf      1.14      1.37  21.355  <.0001

Results are averaged over the levels of: Deviant_threshold 
Results are given on the logit (not the response) scale. 
Confidence level used: 0.95 

$contrasts
 contrast   estimate     SE  df asymp.LCL asymp.UCL z.ratio p.value
 high - low    0.113 0.0828 Inf   -0.0495     0.275   1.363  0.1730

Results are averaged over the levels of: Deviant_threshold 
Results are given on the log odds ratio (not the response) scale. 
Confidence level used: 0.95 
Similarity Plot
Similarity Analysis
Type III Analysis of Variance Table with Satterthwaite's method
                                         Sum Sq Mean Sq NumDF DenDF  F value
targetpair                                 1520    1520     1   458   5.5428
Deviant_threshold                         70303   70303     1   458 256.4093
matrix_cond                                   9       9     1   458   0.0312
targetpair:Deviant_threshold              83118   83118     1   458 303.1459
targetpair:matrix_cond                      839     839     1   458   3.0601
Deviant_threshold:matrix_cond                14      14     1   458   0.0511
targetpair:Deviant_threshold:matrix_cond   1593    1593     1   458   5.8116
                                          Pr(>F)    
targetpair                               0.01898 *  
Deviant_threshold                        < 2e-16 ***
matrix_cond                              0.85992    
targetpair:Deviant_threshold             < 2e-16 ***
targetpair:matrix_cond                   0.08091 .  
Deviant_threshold:matrix_cond            0.82134    
targetpair:Deviant_threshold:matrix_cond 0.01631 *  
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
$emtrends
 targetpair Deviant_threshold.trend   SE  df lower.CL upper.CL t.ratio p.value
 DN                         -57.668 2.60 458   -62.78   -52.55 -22.155  <.0001
 NN                          -0.899 2.29 458    -5.39     3.59  -0.393  0.6945

Results are averaged over the levels of: matrix_cond 
Degrees-of-freedom method: satterthwaite 
Confidence level used: 0.95 

$contrasts
 contrast estimate   SE  df lower.CL upper.CL t.ratio p.value
 DN - NN     -56.8 3.26 458    -63.2    -50.4 -17.411  <.0001

Results are averaged over the levels of: matrix_cond 
Degrees-of-freedom method: satterthwaite 
Confidence level used: 0.95 
# A tibble: 4 × 14
# Groups:   matrix_cond [2]
  matrix_cond id       term        estimate std.error statistic p.value conf.low
  <chr>       <chr>    <chr>          <dbl>     <dbl>     <dbl>   <dbl>    <dbl>
1 high        below_.5 Deviant_th…    -9.01      6.81    -1.32  0.188     -22.5 
2 high        above_.5 Deviant_th…     5.47      6.75     0.811 0.419      -7.88
3 low         below_.5 Deviant_th…   -19.0       7.21    -2.64  0.00925   -33.3 
4 low         above_.5 Deviant_th…    22.4       6.69     3.34  0.00107     9.13
# ℹ 6 more variables: conf.high <dbl>, r.squared <dbl>, adj.r.squared <dbl>,
#   df <dbl>, df.residual <int>, nobs <int>
ISM Analysis
New Agent Prediction Plot
Prediction Analysis
# A tibble: 4 × 14
# Groups:   matrix_cond [2]
  matrix_cond id       term        estimate std.error statistic p.value conf.low
  <chr>       <chr>    <chr>          <dbl>     <dbl>     <dbl>   <dbl>    <dbl>
1 high        below_.5 Deviant_th…   -23.5       10.5    -2.24   0.0266    -44.2
2 high        above_.5 Deviant_th…    -8.35      10.6    -0.786  0.433     -29.4
3 low         below_.5 Deviant_th…    -5.17      12.2    -0.424  0.672     -29.3
4 low         above_.5 Deviant_th…   -14.2       11.8    -1.21   0.228     -37.5
# ℹ 6 more variables: conf.high <dbl>, r.squared <dbl>, adj.r.squared <dbl>,
#   df <dbl>, df.residual <int>, nobs <int>
Analysis of Variance Table

Response: confidence
                      Df Sum Sq Mean Sq F value  Pr(>F)  
deviance               4   9049 2262.16  3.0597 0.01661 *
matrix_cond            1    700  700.21  0.9471 0.33100  
deviance:matrix_cond   4    898  224.47  0.3036 0.87555  
Residuals            448 331230  739.35                  
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
$emmeans
 deviance emmean   SE  df lower.CL upper.CL t.ratio p.value
 0          61.3 2.92 448     55.6     67.1  20.967  <.0001
 0.25       52.6 2.82 448     47.1     58.1  18.677  <.0001
 0.5        53.8 2.72 448     48.5     59.2  19.798  <.0001
 0.75       50.2 2.91 448     44.5     55.9  17.284  <.0001
 1          48.3 2.94 448     42.5     54.0  16.447  <.0001

Results are averaged over the levels of: matrix_cond 
Confidence level used: 0.95 

$contrasts
 contrast                    estimate   SE  df lower.CL upper.CL t.ratio
 deviance0 - deviance0.25        8.71 4.06 448   -2.410    19.83   2.145
 deviance0 - deviance0.5         7.46 3.99 448   -3.473    18.40   1.869
 deviance0 - deviance0.75       11.09 4.12 448   -0.196    22.38   2.691
 deviance0 - deviance1          13.03 4.14 448    1.685    24.38   3.146
 deviance0.25 - deviance0.5     -1.25 3.92 448  -11.968     9.48  -0.318
 deviance0.25 - deviance0.75     2.38 4.05 448   -8.698    13.47   0.589
 deviance0.25 - deviance1        4.32 4.07 448   -6.818    15.46   1.063
 deviance0.5 - deviance0.75      3.63 3.98 448   -7.270    14.53   0.912
 deviance0.5 - deviance1         5.57 4.00 448   -5.391    16.53   1.392
 deviance0.75 - deviance1        1.94 4.13 448   -9.372    13.25   0.470
 p.value
  0.2029
  0.3356
  0.0568
  0.0151
  0.9978
  0.9766
  0.8254
  0.8922
  0.6334
  0.9900

Results are averaged over the levels of: matrix_cond 
Confidence level used: 0.95 
Conf-level adjustment: tukey method for comparing a family of 5 estimates 
P value adjustment: tukey method for comparing a family of 5 estimates 
Moderator: Last Opinion
0
(N=89)
0.25
(N=95)
0.5
(N=100)
0.75
(N=88)
1
(N=86)
Overall
(N=458)
pred_maj
Yes 9 (10.1%) 18 (18.9%) 12 (12.0%) 16 (18.2%) 19 (22.1%) 74 (16.2%)
No 80 (89.9%) 77 (81.1%) 88 (88.0%) 72 (81.8%) 67 (77.9%) 384 (83.8%)
# A tibble: 4 × 14
# Groups:   pred_maj [2]
  pred_maj id      term  estimate std.error statistic p.value conf.low conf.high
  <lgl>    <chr>   <chr>    <dbl>     <dbl>     <dbl>   <dbl>    <dbl>     <dbl>
1 FALSE    below_… Devi…   -20.7       8.14    -2.55   0.0115    -36.8     -4.70
2 FALSE    above_… Devi…    -4.17      8.71    -0.479  0.632     -21.3     13.0 
3 TRUE     below_… Devi…    38.4      24.9      1.54   0.131     -12.0     88.8 
4 TRUE     above_… Devi…   -40.7      19.0     -2.14   0.0379    -79.1     -2.38
# ℹ 5 more variables: r.squared <dbl>, adj.r.squared <dbl>, df <dbl>,
#   df.residual <int>, nobs <int>
Analysis of Variance Table

Response: confidence
                   Df Sum Sq Mean Sq F value    Pr(>F)    
deviance            4   9049  2262.2  3.1999 0.0131279 *  
pred_maj            1   9735  9735.5 13.7713 0.0002324 ***
deviance:pred_maj   4   6384  1595.9  2.2574 0.0621099 .  
Residuals         448 316709   706.9                      
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
0
(N=89)
0.25
(N=95)
0.5
(N=100)
0.75
(N=88)
1
(N=86)
Overall
(N=458)
pns_med
High 37 (41.6%) 47 (49.5%) 42 (42.0%) 44 (50.0%) 37 (43.0%) 207 (45.2%)
Low 51 (57.3%) 48 (50.5%) 58 (58.0%) 44 (50.0%) 49 (57.0%) 250 (54.6%)
Missing 1 (1.1%) 0 (0%) 0 (0%) 0 (0%) 0 (0%) 1 (0.2%)
# A tibble: 4 × 14
# Groups:   pns_med [2]
  pns_med id       term  estimate std.error statistic p.value conf.low conf.high
  <chr>   <chr>    <chr>    <dbl>     <dbl>     <dbl>   <dbl>    <dbl>     <dbl>
1 High    below_.5 Devi…   -9.68      13.6   -0.714    0.476     -36.5    17.1  
2 High    above_.5 Devi…   -0.121     13.5   -0.00893  0.993     -26.8    26.6  
3 Low     below_.5 Devi…  -18.8        9.59  -1.96     0.0520    -37.7     0.165
4 Low     above_.5 Devi…  -20.1        9.39  -2.14     0.0341    -38.6    -1.53 
# ℹ 5 more variables: r.squared <dbl>, adj.r.squared <dbl>, df <dbl>,
#   df.residual <int>, nobs <int>
Analysis of Variance Table

Response: confidence
                  Df Sum Sq Mean Sq F value  Pr(>F)  
deviance           4   9170 2292.56  3.1300 0.01477 *
pns_med            1      6    6.18  0.0084 0.92688  
deviance:pns_med   4   5287 1321.71  1.8045 0.12685  
Residuals        447 327402  732.44                  
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Order of deviant across rounds
Opinion Round
0
(N=458)
1
(N=458)
2
(N=458)
3
(N=458)
4
(N=458)
5
(N=458)
6
(N=458)
7
(N=458)
Overall
(N=3664)
trialnum
0 52 (11.4%) 68 (14.8%) 68 (14.8%) 54 (11.8%) 49 (10.7%) 77 (16.8%) 56 (12.2%) 56 (12.2%) 480 (13.1%)
1 47 (10.3%) 53 (11.6%) 42 (9.2%) 66 (14.4%) 57 (12.4%) 67 (14.6%) 58 (12.7%) 56 (12.2%) 446 (12.2%)
2 59 (12.9%) 61 (13.3%) 56 (12.2%) 51 (11.1%) 65 (14.2%) 45 (9.8%) 63 (13.8%) 51 (11.1%) 451 (12.3%)
3 63 (13.8%) 59 (12.9%) 58 (12.7%) 53 (11.6%) 53 (11.6%) 49 (10.7%) 51 (11.1%) 45 (9.8%) 431 (11.8%)
4 64 (14.0%) 46 (10.0%) 61 (13.3%) 69 (15.1%) 66 (14.4%) 59 (12.9%) 51 (11.1%) 68 (14.8%) 484 (13.2%)
5 47 (10.3%) 62 (13.5%) 61 (13.3%) 59 (12.9%) 59 (12.9%) 62 (13.5%) 60 (13.1%) 65 (14.2%) 475 (13.0%)
6 66 (14.4%) 52 (11.4%) 54 (11.8%) 46 (10.0%) 59 (12.9%) 60 (13.1%) 63 (13.8%) 59 (12.9%) 459 (12.5%)
7 60 (13.1%) 57 (12.4%) 58 (12.7%) 60 (13.1%) 50 (10.9%) 39 (8.5%) 56 (12.2%) 58 (12.7%) 438 (12.0%)
Things to note
  • The PNS moderator is a median split
Unresolved
  • all good